System and method for controlling user repeatability and reproducibility of automated image annotation correction

ABSTRACT

Systems and methods are disclosed for controlling image annotation. One method includes acquiring a digital representation of image data and generating a set of image annotations for the digital representation of the image data. The method also may include determining an association between members of the set of image annotations and generating one or more groups of members based on the association. A representative annotation from the one or more groups may also be determined, presented for selection, and the selection may be recorded in memory.

RELATED APPLICATION

This application claims the benefit of priority from U.S. ProvisionalApplication No. 61/882,512, filed Sep. 25, 2013, which is herebyincorporated herein by reference in its entirety.

FIELD

Various embodiments of the present disclosure relate generally tomedical imaging and related methods. More specifically, particularembodiments of the present disclosure relate to systems and methods forcontrolling user repeatability and reproducibility of automated imageannotation correction.

BACKGROUND

Automated image annotation plays an increasing role in commercialsystems. In particular, the medical imaging community reliesincreasingly on the automated analysis and annotation of large images.Since this automated image analysis may be used to drive patient caredecisions, it can be important for the automated results to be validatedand appropriately corrected (if necessary) by a knowledgeable user. Theuser may be a medical doctor, technician, or another individual trainedto evaluate medical images and use the software. Outside of a medicalcontext, the user may be anyone who evaluates the image annotations,including either professionals or consumers.

In a commercial context, a particular concern is the ability of a userto provide corrections of automated image annotations, which arerepeatable and reproducible from one user to another or from a user tothemselves. Particularly in a medical context, diagnosis and treatmentdecisions may be made on the basis of the (corrected) medical imageannotations, so it is important to minimize any dependence of the imageannotation corrections on the user. Manual tools for correctingautomated image annotations (e.g., picking points, drawing lines,drawing shapes (e.g. polygons or circles), modifying pixel labels, usingfree text to adjust annotation labels) are subject to substantial levelsof inter-user and intra-user variability, which should be controlled inorder to provide a reliable image annotation system that maximizespatient benefit.

Thus, a need exists for systems and methods to decrease variability andcontrol user repeatability and reproducibility of automated imageannotations.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of thedisclosure.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for controlling image annotations using a computersystem. One method includes using a processor for acquiring a digitalrepresentation of image data; generating a set of image annotations forthe digital representation of the image data; determining an associationbetween members of the set of image annotations and generating one ormore groups of members based on the associations; determining arepresentative annotation from each of the one or more groups;presenting the one or more representative annotations for selection; andrecording the selection and saving the one or more representativeannotations in memory.

According to certain additional aspects of the present disclosure, onemethod for controlling image annotations using a computer system, mayinclude one or more of: further comprising the following steps prior tothe step of generating a set of image annotations for the digitalrepresentation of the image data; presenting an automated annotation ofthe digital representation for validation and/or correction, andreceiving a user input of an image annotation based on the user'sevaluation of the automated annotation; wherein the image data is of atarget object; wherein the target object is a portion of a body organ;wherein the digital representation of the image data is acquiredelectronically via a network; wherein the set of alternative imageannotations are electronically automatically generated; wherein the setof alternative image annotations are generated using multiple imageanalysis algorithms; wherein the association between members of the setof alternative image annotations is determined by assigning a similarityscore between the members; wherein the association between members ofthe set of alternative image annotations is determined by assigningalternative image annotations to a group if the alternatives are thesame; wherein the automated image annotation comprises a polygon (e.g. abox) around a target feature; and further comprising applying a k-meansalgorithm to a centroid of the box.

In accordance with another aspect, disclosed is a system of controllingimage annotations, the system comprises: a data storage device storinginstructions for controlling image annotation; and a processorconfigured to execute the instructions to perform a method including:acquiring a digital representation of image data; generating a set ofimage annotations for the digital representation of the image data;determining an association between members of the set of imageannotations and generating one or more groups of members based on theassociations; determining a representative annotation from each of theone or more groups; presenting the one or more representativeannotations for selection; and recording the selection and saving theone or more representative annotations in memory.

According to certain additional examples of the present disclosure, onesystem of controlling image annotation, may include one or more of thefollowing aspects: further comprising the following steps prior to thestep of generating a set of image annotations for the digitalrepresentation of the image data: presenting an automated annotation ofthe digital representation for validation and/or correction, andreceiving a user input of an image annotation based on the user'sevaluation of the automated annotation; wherein the image data is of atarget object; wherein the processor is further configured toelectronically acquire the digital representation of image data via anetwork; wherein the processor is further configured to automaticallygenerate the set of alternative image annotations; wherein the processoris further configured to automatically generate the set of alternativeimage annotation using multiple image analysis algorithms; wherein theprocessor in configured to determine the association between members ofthe set of alternative image annotations by assigning a similarity scorebetween the members; wherein processor is configured to determine theassociation between members of the set of alternative image annotationsby assigning alternative Image annotations to a group if thealternatives are the same; and wherein the processor is configured togenerate a box around a target feature.

In accordance with yet another embodiment, a non-transitory computerreadable medium for use on a computer system containingcomputer-executable programming instructions for controlling imageannotations, the instructions being executable by the computer systemfor: acquiring a digital representation of image data; generating a setof image annotations for the digital representation of the image data;determining an association between members of the set of imageannotations and generating one or more groups of members based on theassociations; determining a representative annotation from each of theone or more groups; presenting the one or more representativeannotations for selection; and recording the selection and saving theone or more representative annotations in memory.

According to certain additional examples of the present disclosure, onenon-transitory computer-readable medium for use on a computer systemcontaining computer-executable programming instructions for controllingimage annotations, the instructions being executable by the computersystem for determining an association between members of the set ofalternative image annotations and generating one or more groups ofmembers based on the associations, wherein the association betweenmembers of the set of alternative image annotations is determined byassigning a

Additional objects and advantages of the disclosed embodiments will beset forth in part in the description that follows, and in part will beapparent from the description, or may be learned by practice of thedisclosed embodiments. The objects and advantages of the disclosedembodiments wilt be realised and attained by means of the elements andcombinations particularly pointed out in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 is a block diagram of an exemplary system and network forcontrolling user repeatability and reproducibility of automated imageannotation correction.

FIG. 2 is a block diagram of an exemplary method for presentingalternative image annotations to a user, according to an exemplaryembodiment of the present disclosure.

FIG. 3 is a block diagram of exemplary options for generating a set ofalternative image annotations, according to an exemplary embodiment ofthe present disclosure.

FIG. 4 is a block diagram of exemplary options for determining anassociation between members of a set of alternative image annotations.

FIG. 5 is a block diagram of an exemplary method for presentingalternative feature detection candidates to a user, according to anexemplary embodiment of the present disclosure.

FIG. 6 is a block diagram of an exemplary method for presentingalternative image segmentation candidates to a user, according to anexemplary embodiment of the present disclosure.

FIG. 7 is a block diagram of an exemplary method for hierarchal imageannotation selection, according to an exemplary embodiment of thepresent disclosure.

FIG. 8 is a simplified block diagram of an exemplary computer system inwhich embodiments of the present disclosure may be implemented.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

As described above, correction of digital images may be desired toprovide a clearer or more accurate depiction of a target object.

The present disclosure is directed to systems and methods forcontrolling user repeatability and reproducibility for automated imageannotation correction. The systems and methods involve presenting theuser with various alternative annotations of a digital image of a targetobject. The user may select one or more of the alternative annotationsto validate an automated annotation and/or correct an erroneous imageannotation. Therefore, instead of directly correcting an automated imageannotation, the user may be presented with alternative solutions thatmay be selected to correct the automated image annotation. This processof correcting an image annotation by selection from alternatives ratherthan the user directly correcting the annotation may dramaticallyimprove the reproducibility and repeatability of the correction processby reducing variability in user annotations and may increaseefficiencies in obtaining a correct image annotation. In addition, theuser selections may be processed and used by suitable data analysistools to improve the automated image annotations so as to produce moreaccurate results.

Referring now to the figures, FIG. 1 depicts a block diagram of anexemplary system and network for controlling user repeatability andreproducibility of automated image annotation correction. Specifically,FIG. 1 depicts a plurality of physicians of other healthcareprofessionals 102 and third party providers 104, any of whom may beconnected to an internal or external electronic network 100, such as theInternet, through one or more computers, servers, and/or handheld mobiledevices. Physicians 102 and/or third party providers 104 may create orotherwise obtain images of one or more target regions of one or morepatients. For example, images of one or more patients' cardiac systems,vascular systems, neurological systems, etc., may be created or obtainedby any suitable imaging devices. The physicians 102 and/or third partyproviders 104 may also obtain any combination of patient-specificinformation, such as age, medical history, blood pressure, bloodviscosity, etc. This information may be directly obtained by examinationof the patient or retrieved from the patient's medical record, such byelectronically accessing the patient's electronic medical record. Insome examples, the third party provider 104 may be an imaging laboratoryand the patient-specific information (e.g. medical history, currenttreatment plan, family history) may be electronically retrieved (e.g.automatically) via the electronic network 100, e.g. via the Internet.Physicians 102 and/or third party providers 104 may transmit the images,such as cardiac and/or vascular images, and/or patient-specificinformation to server systems 106 over the electronic network 100. Theelectronic network 100 may include various security components forvalidating and securing the privacy of the patient-specific informationtransmitted and complying with all pertinent laws and regulations. Forexample, the electronic network 100 may encrypt or encode the patient'sidentity by removing any identifying information (e.g. patient name,address, social security number, phone number, etc.,) and replacing thisinformation with an identification code that is generated, transmitted,and saved in digital memory. In this and any other suitable manner thatcomplies with applicable health information patient privacy rules, theprivacy of the patient may be maintained. In addition, the digital imagerepresentations also may be encrypted or encoded.

This electronic transmission of data may occur at any suitable time. Forexample, the images may be electronically locally saved on one or moredigital storage devices at the physician 102 or third party provider 104and the data, which may be electronically encrypted, may be transmittedat a later time, or the data may be transmitted as soon as theinformation is obtained. In some embodiments, portions of the data maybe transmitted at different times. For example, the patient-specificinformation may be transmitted before or after the image is transmitted.The data may be encoded in any suitable manner and may be decoded orparsed using one or more processors having program instructionsconfigured to electronically/digitally decode/decrypt and/or parse thedata. Server systems 106 may include storage devices for storing imagesand data received from physicians 102 and/or third party providers 104.Server systems 106 also may include processing devices (e.g. one or anetwork of multiple processors) for processing images and data stored inthe storage devices.

FIG. 2 is a block diagram of an exemplary method 200 for controllinguser repeatability and reproducibility for automated image annotationcorrection. Method 200 may be performed by the server systems 106, basedon information, images, and data received from physicians 102 and/orthird party providers 104 over the electronic network 100. The method200 may include various steps 202-214 which may result in the userproviding electronic feedback, such as validation and/or correction ofan automated image annotation by making selections from a group ofalternative image annotations. This may improve the repeatability andreproducibility of automated image annotation correction.

The method 200 may include acquiring a digital representation of imagedata at step 202. The image may be of any suitable target, such as anorgan or portion of a patient's body, or any other suitable target. Forexample, the target may be a portion of the heart, lungs, brain, etc.The image of the target may be obtained using any suitable technology,such as an electronic scan from computed tomography, magnetic resonanceimages, ultrasound images, images from a 3-D scanner, etc.

In one embodiment, the method 200 may be used for modeling andsimulation for blood flow of the heart (Taylor, Fonte, & Min,“Computational Fluid Dynamics Applied to Cardiac Computed Tomography forNoninvasive Quantification of Fractional Flow Reserve” J Am CollCardiol. 2013;61(22):2233-2241 the disclosure of which is incorporatedherein by reference in its entirety), which may involve an extremelyprecise image segmentation from a cardiac CT image to create apatient-specific 3-D geometric model of the patient's heart. This 3-Dmodel may be validated and corrected by trained users either local orremote to server systems 106, by using the methods disclosed herein inorder to control the repeatability and reproducibility of automatedimage annotation correction and ensure the accuracy of the blood flowsimulations and the treatment decisions derived from the simulation.

The digital representation of the image may be acquired at step 202. Thedigital representation of the image may have been previously transmittedby the physician 102 or the third party provider 104 to an electronicstorage device. The digital representation electronically transmitted bythe physician 102 of the third party provider 104 may be automaticallyelectronically encrypted. The digital representation of the image maythen be retrieved from the digital storage, such as a hard drive,network drive, remote server, etc. of any suitable computational device,such as by a connected computer, laptop, digital signal processor (DSP),server etc., and may be automatically electronically decrypted. Thedigital representation of the image may be retrieved from the memory ordigital storage in any suitable manner. For example, a computationaldevice may include programming with instructions to request the digitalrepresentation from the memory or the digital storage at a certain timeand send it to the computational device of the user, such as atechnician, at a certain time.

The digital representation of the image data acquired at step 202 maythen be processed by one or more computational devices. In someembodiments, a set of image annotations may be directly generated forthe digital representation of the image data at step 206. Alternatively,following the acquisition of the digital representation of image data atstep 202, an automated annotation of the digital representation may bepresented to the user for validation and or correction at step 204 andthe user's input at step 205 may be automatically electronicallyprocessed. The term image annotation can generically represent anyidentification in an image (including, but not limited to, a 2-D image,a 3-D image, or an image of greater dimension) that includes alocalization or a labeling. Examples may include: localization ofparticular points (landmarks), localization of lines (e.g., diameters,centerlines), 2-D regions of interest (a 2-D segmentation), 3-D regionsof interest (a 3-D segmentation), an n-D region of interest (an n-Dsegmentation), an n-D+time region of interest (an n-D segmentationtracked through time) or a labeling of one or more identifiedstructures.

The automated image annotation at step 204 may be generated by applyingan automated image analysis system. The automated image analysis systemmay be any suitable automated image analysis system and may includevarious annotations. In one example, the automated image analysis systemmay annotate a 3-D segmentation of the aorta (e.g., using (Kirisli, etat, “Fully automatic cardiac segmentation from 3D CTA data: amulti-atlas based approach,” Proc. of SPIE Vol. 7623 762305-9, 2010),the disclosure of which is incorporated herein by reference in itsentirety). In another example, the automated image analysis system mayannotate the location of ostial points (e.g., using (Zheng, Tek,Funka-Lea, Zhou, Vega-Higuera, & Comaniciu, “Efficient Detection ofNative and Bypass Coronary Ostia in Cardiac CT Volumes: Anatomical vs.Pathological Structures,” Proc. Int'l Conf. Medical Image Computing andComputer Assisted Intervention, 2011) the disclosure of which isincorporated herein by reference in its entirety). In another example,the automated image analysis system may annotate the location of theaortic valve point (e.g., using (Zheng, Barbu, Georgescu, Scheuering, &Comaniciu, “Four-Chamber Heart Modeling and Automatic Segmentation for3D Cardiac CT Volumes Using Marginal Space Learning and SteerableFeatures,” IEEE Transactions on Medical Imaging, Vol. 27, No. 11, pp.1688-1681, 2008), the disclosure of which is incorporated herein byreference in its entirety). In another example, the automated imageanalysis system may annotate coronary vessel centerlines (e.g., using(Kitamura, Li, & Ito, “Automatic coronary extraction by superviseddetection and shape matching” Biomedical Imaging (ISBI), 2012 9th IEEEinternational Symposium on May 2-5 2012), the disclosure of which isincorporated herein by reference in its entirety). In another example,the automated image analysis system may include labeling of the vesselcenter lines by vessel name (i.e., RCA, LAD, and LCX). This labeling maybe performed by using a set of training labels to determine thestatistics of the geometric positioning of each labeled vesselcenterline and assigning the vessel centerline the labels having themaximum likelihood based on its geometric position (see e.g., (Lorenz &Berg, “A Comprehensive Shape Model of the Heart” Medical Image Analysis10 (2006) 657-670, 18 May 2006), the disclosure of which is incorporatedherein by reference in its entirety). In another example, the automatedimage analysis system may annotate 3-D segmentation of the coronaryvessel lumen (e.g., using (Schaap, et al. “Robust Shape Regression forSupervised Vessel Segmentation and its Application to CoronarySegmentation in CTA” IEEE Transactions on Medical Imaging, Vol. 30, No.11, November 2011), the disclosure of which is incorporated herein byreference in its entirety). In another example, the automated imageanalysis system may annotate a 3-D segmentation of the left ventriclemyocardium (e.g., using (Kirisli, et al., 2010)). Any of the aboveexamples may be used individually or in combination.

The image annotation may be presented to the user in any suitablemanner, for example the image annotation may be sent via a network tothe user's computational device for presentation on an output, such as amonitor, tablet, etc. The image annotation may have any suitable form.For example, the image annotation may be an interactive digital imagepresented via a graphical user interface (GUI) with various selectableportions for the user to select or deselect in any suitable manner,either by actively selecting or deselecting various portions or doing sopassively (e.g. rapidly scrolling over a portion that is acceptable orunacceptable according to any pre-selected user preferences orsettings).

The user may review the automated image annotation and provideinput/feedback on the image annotation at step 205, such as validationof the image annotation and/or possible correction. The user may providefeedback in any suitable manner, such as by using a GUI. For example,the user may select or deselect various portions of the image annotationto indicate validation and/or correction.

In some embodiments, the image annotation presented to the user mayinclude one or more selectable boxes or icons representing various imageannotations. The annotations may be presented to the user in anysuitable manner. Some of the annotations may change or be dependent onother annotations. As such, if one of the annotations with a dependencyis modified by the user upon review, the automated algorithms forconsequent annotations may be re-run with the modified annotation asinput. For example, the annotations may be presented to the user in theorder:

-   a. Aorta segmentation-   b. Ostia points-   e. Aortic valve point-   d. Left ventricle myocardium segmentation-   e. Coronary vessel centerlines-   f. Vessel labeling-   g. Vessel lumen segmentation

Based on the input/feedback received from the user on the automatedimage annotation at step 205 presented to the user at 204, a set ofalternative image annotations may be generated at step 206. The feedbackmay be processed and stored for later use. In some aspects, the userfeedback data may be added or aggregated with other user feedback forprocessing and improving the accuracy of the automated image annotation,using various algorithms, such as machine language algorithms.

The alternative image annotations may be generated in any suitablemanner. Examples of alternative image annotations 300 are shown in FIG.3, and may include using one or more automated image analysisalgorithm(s)/systems 302. Examples of automated image analysisalgorithm(s)/systems include: face detection in a digital camera, imageor video labeling (tagging) In a collection of images/videos (e.g.,internet videos), 3-D organ segmentation in CT medical images forradiation therapy planning, 2-D left ventricle segmentation inultrasound medical images for computing an ejection fraction, 2-D celldetection in a microscopy image for image-based cytometry, 2-D celltracking through video in an optical microscopy image for determinationof mitosis events, 3-D tumor segmentation and feeder vessel centerlineannotation in CT medical images for chemoembolization planning, 3-D bonefracture segmentation in CT medical images for bone reconstructionplanning, and/or tumor detection and segmentation in a digitalmammography application. Another alternative image annotation, as shownin FIG. 3, may include perturbing the image 304 (e.g. via noise, imageenhancement, resampling, rotation, scaling, filtering, etc.). Automatedimage analysis algorithm(s)/systems with a variation of internalparameters 306 may also be used.

In some examples in which an automated image annotation is presented tothe user (step 204) prior to generating a set of image annotations forthe digital representation of the image data (step 206), the imageannotations generated in step 206 maybe alternative image annotationsbased on input received from the user at step 205. The set of imageannotations (which may be alternative image annotations based on inputreceived from the user at step 205) generated at 206 may be processedand an association between members of the set of image annotations (oralternative image annotations based on input received from the user) maybe determined at step 208 by using any suitable methods. An example ofsuch a method 400 is shown in FIG. 4, and may include assigning asimilarity score between members of a group or cluster of alternativeimage annotations and/or using a clustering method to determine clustersor groups at step 402. Any suitable algorithm or multiple algorithms maybe used to cluster the alternative image annotations, such as by, forexample, using hierarchical clustering, and centroid models (e.g.k-means algorithm), etc. In addition or alternatively, an associationbetween the members of the set of alternative image annotations may bedetermined by assigning alternative image annotations to a group if thealternatives are similar at step 404 and/or members of some otherequivalence class.

A representative annotation from one or more groups may be determined atstep 210. One or more of the representative annotations for selectionmay be presented at 212. The annotations may be presented to the uservia a GUI or using any other suitable method. The user selection from212 may be recorded and the image saved at 214 in digital memory. Insome aspects, the user selection recorded at step 214 may be processedusing various algorithms (e.g. machine learning algorithms) to improvethe accuracy of subsequent automated annotations and may be digitallyidentified, (e.g. using a tag) for use in analyzing user selectionbehavior. For example, the user selections may be identified by userdemographic (e.g. level of experience, attendance of training courses,geographic location), patient information (e.g. chronic disease patient,patient age, gender, etc.), and various other information.

According to one example, fifty different centerlines may be generatedat step 206, and each of the centerlines may be processed and assigned asimilarity score at step 208, the basis of which may be used to groupsimilar center lines (e.g. centerlines 1-7 and 15-25 may be similar andassigned group A, centerlines 8-14 and 26-35 may be similar and assignedgroup B, and centerlines 36-50 may be similar and assigned group C). Arepresentative centerline from each group A, B and C, (e.g. centerline1, 8, and 26) may be determined at 210 and presented to the user at step212.

In some aspects, the user selection recorded at step 214 may be added toa database of other user selections and processed to determine anyassociations between the user, patients, and/or any other similaritiesto improve the accuracy of the automated annotation method, e.g. usingany identifying information. For example, it may be determined thatusers who were trained during a certain time period all make similarselections compared to other users trained at different times, and themethod may further include steps to make recommendations to users toimprove user accuracy and uniformity in selecting the annotations.

In another embodiment a similar method as described above and shown inFIG. 2 is shown in FIG. 5. FIG. 5 shows an example of a method 500,which includes applying an automated image analysis system to produce aset of feature detection candidates. The method 500 may includeacquiring a digital representation of image data, such as a digitalrepresentation of a patient's heart, scans in step 502, in a similarmanner as described in step 202 above. For example, a digitalrepresentation of a patient's heart scan, such as a 3-D cardiac computedtomography (CT) image, may be acquired from the memory or digitalstorage of a computational device, such as a computer.

The digital representation acquired at step 502 may be analyzed andprocessed by an automated image analysis system to produce a set offeature detection candidates at step 504. The feature detectioncandidates may be any suitable target location(s) on the image, such as,for example a certain point or portion of the image, an object (e.g. ananatomical landmark, organ, etc.). Each feature detection candidate maybe described on the GUI in any suitable manner. For example, a box,circle, or any other shape may surround the target feature. The targetfeature may be detected in any suitable manner, such as, for example, byusing a Viola-Jones detection system and/or any other suitable featuredetection system. The automated image analysis system may include analgorithm, which includes assigning each feature detection box aresponse score indicating the certainty of the algorithm to the positionand size of the box bounding the detected object.

At step 506, a set of alternative detection candidates may be generatedin any suitable manner. In one embodiment, the assigned response scorefor each feature detection calculated in step 504 may be compared to apre-determined threshold score. The pre-determined threshold score maybe adjusted based one or more suitable factors. The set of alternativedetections may be generated based on feature detections having anassigned response score equal to or above the pre-determined threshold.

The alternative detection candidates generated in step 506 may befurther processed into groups in any suitable manner in step 508. Forexample, a k-means algorithm or other suitable algorithm may be appliedto the centroids of each bounding box of each of the alternativedetection candidates.

A representative detection from each group may be determined at step510. For example, a detection from each group may be selected for whichthe centroid of the representative is closest (e.g. by Euclideandistance) to the centroid of each group. The representative detectionsfrom each group determined at step 510 may be presented to the user atstep 512. The user may be permitted to make one or more selections fromthe representative detection in any suitable manner. The userselection(s) may be saved at step 514 to an electronic storage medium(e.g. hard drive, computer RAM, network communication channel, etc.)

In another embodiment, another similar method as described above andshown in FIGS. 2 and 5 is shown in FIG. 6, which shows a method 600 inwhich image segmentation may be used to automatically correct imageannotations. A digital representation of a target object, such as apatient's heart may be acquired at step 602 in any suitable manner, asdescribed above with reference to FIGS. 2 and 5. An automated imageanalysis system may be applied to produce a set of image segmentationcandidates at step 604, for example, by using level set, graph basedmethods (e.g., graph cuts, random walker), active shape models, and/oractive appearance models.

A set of alternative segmentation candidates may be generated at step606 using any suitable method. Each method may include severalparameters. A set of alternative segmentations may therefore begenerated by randomly selecting several parameter sets, applying severalimage segmentation techniques, and perturbing the input image by rigidtransformation and the addition of Gaussian noise to the voxels.

A Dice coefficient may be computed at step 608 between all or a portionof pairs of alternative segmentations. Groups of alternativesegmentations may be determined by applying a clustering algorithm onthe set of alternative segmentations in which the reciprocal of the Dicecoefficient may be interpreted as a distance between segmentations.Examples of clustering algorithms include spectral partitioning andisoperimetric portioning.

A representative segmentation from each group determined in step 608 maybe determined at step 610. The representative segmentation may bedetermined by randomly selecting a segmentation from each group, bycreating an average segmentation from the segmentations in the group,and/or by selecting the segmentation from the group that is closest(e.g. has the highest Dice overlap) to the average segmentation.

The representative segmentations determined at step 610 may be presentedto the user in any suitable manner, for example, via the GUI. The usermay then make a selection from among the representative segmentations atstep 612.

The user selection at 612 may be recorded and the correspondingrepresentative segmentation may be saved to an electronic storagemedium.

In another embodiment, as shown in FIG. 7, a hierarchal selection method700 may be used to more precisely specify which automatically generatedannotation is selected, instead of selecting the automaticallydetermined representative example from one of the groups. The method 700may be used in conjunction with other methods, such as the featuredetection method 500 and image segmentation method 600, as describedabove. Similar to the method 200 described above, the hierarchal method700 may include acquiring a digital representation of a target object,such as a patient's heart scan at step 702, and producing a set of imageannotations at step 704.

The image annotations may be clustered at step 706 in any suitablemanner, such as using by applying a k-means algorithm and/or determininga representative detection as described above in FIG. 5 steps 508-510. Arepresentative example from each cluster may be presented at step 708 inany suitable manner as described in FIG. 5 step 512. The exampleselected by -the user from step 708 may be recorded on a digital storagemedium in step 710. Steps 706-710 may be repeated one or more times.Each time the steps 706-710 are repeated, the step may only start withthe image annotation that belong to the cluster that was selected. Inaddition, the image annotations that fall in the same cluster may begenerated, and the representative segmentations that were selected inthe last iteration may be stored in an electronic storage medium in step712.

It should be appreciated that any type of computing system, such as acomputer having a digital memory storage device, a processor, and anydesired user interlaces may be configured to perform the presentlydisclosed methods. Moreover, any type of servers, clustered computers,and/or cloud computers may be configured to perform the presentlydisclosed methods. Any of the above referenced devices may be connectedto a network, such as the Internet, to receive and transmit data used inperforming the presently disclosed methods.

FIG. 8 is a simplified block diagram of an exemplary computer system 800in which embodiments of the present disclosure may be implemented, forexample as any of the physician devices or severs 102, third partydevices or servers 104, and server systems 106. A platform for a server800, for example, may include a data communication interface for packetdata communication 860. The platform may also include a centralprocessing unit (CPU) 820, in the form of one or more processors, forexecuting program instructions. The platform typically includes aninternal communication bus 810, program storage, and data storage forvarious data files to be processed and/or communicated by the platformsuch as ROM 830 and RAM 840, although the server 800 often receivesprogramming and data via a communications network (not shown). Thehardware elements, operating systems, and programming languages of suchequipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. The server 800also may include input and output ports 850 to connect with input andoutput devices such as keyboards, mice, touchscreens, monitors,displays, etc. Of course, the various server functions may beimplemented in a distributed fashion on a number of similar platforms,to distribute the processing load. Alternatively, the servers may beimplemented by appropriate programming of one computer hardwareplatform.

As described above, the computer system 800 may include any type orcombination of computing systems, such as handheld devices, personalcomputers, servers, clustered computing machines, and/or cloud computingsystems. In one embodiment, the computer system 800 may be an assemblyof hardware, including a memory, a central processing unit (“CPU”),and/or optionally a user interface. The memory may include any type ofRAM or ROM embodied in a physical storage medium, such as magneticstorage including floppy disk, hard disk, or magnetic tape;semiconductor storage such as solid-state disk (SSD) or flash memory;optical disc storage; or magneto-optical disc storage. The CPU mayinclude one or more processors for processing data according toinstructions stored in the memory. The functions of the processor may beprovided by a single dedicated processor or by a plurality ofprocessors. Moreover, the processor may include, without limitation,digital signal processor (DSP) hardware, or any other hardware capableof executing software. The user interface may include any type orcombination of input/output devices, such as a display monitor,touchpad, touchscreen, microphone, camera, keyboard, and/or mouse.

Program aspects of the technology may be thought of as “products” or“articles of manufacture” typically in the form of executable codeand/or associated data that is carried on or embodied in a type ofmachine-readable medium. “Storage” type media include any or all of thetangible memory of the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide non-transitory storage atany time for the software programming. All or portions of the softwaremay at times be communicated through the Internet or various othertelecommunication networks. Such communications, for example, may enableloading of the software from one computer or processor into another, forexample, from a management server or host computer of the mobilecommunication network into the computer platform of a server and/or froma server to the mobile device. Thus, another type of media that may bearthe software elements includes optical, electrical and electromagneticwaves, such as used across physical interfaces between local devices,through wired and optical landline networks and over various air-links.The physical elements that carry such waves, such as wired or wirelesslinks, optical links, or the like, also may be considered as mediabearing the software. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms, such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

1-20. (canceled)
 21. A computer-implemented method of controlling imageannotation, using at least one computer system, the method comprisingusing a processor for: generating, by the at least one computer system,a plurality of image annotations of anatomic features identified withina digital representation of image data; presenting, by the at least onecomputer system, one or more image annotations among the plurality ofimage annotations of anatomic features to a user; receiving feedbackfrom the user regarding the one or more image annotations; based on thereceived feedback, perturbing the digital representation of image data;generating, by the at least one computer system, alternative imageannotations of anatomic features identified within the perturbed digitalrepresentation of image data; and presenting, by the at least onecomputer system, the alternative image annotations for selection orcorrection by a user.
 22. The computer-implemented method of claim 21,further comprising: generating one or more groups of image annotations,each group comprising one or more image annotations among the generatedplurality of image annotations; determining a representative annotationfor each group selected from among the image annotations of the group,the representative annotation being determined as an annotation amongthe image annotations of the group closest to an aggregation of theimage annotations of the group; and presenting the representativeannotation of each group of the one or more groups for selection orcorrection by a user.
 23. The computer-implemented method of claim 21,further comprising: determining, by the at least one computer system,one or more associations between members of a set of image annotationsof the plurality of image annotations, wherein each group of imageannotations is generated based on the determined one or moreassociations.
 24. The computer-implemented method of claim 21, whereinthe digital representation of image data is acquired electronically viaa network.
 25. The computer-implemented method of claim 22, wherein therepresentative annotation identifies anatomic features identified withinthe image data.
 26. The computer-implemented method of claim 21, whereinthe plurality of image annotations are generated using multiple imageanalysis algorithms.
 27. The computer-implemented method of claim 23,wherein each of the one or more associations between members of the setof image annotations is determined by assigning a similarity scorebetween the members.
 28. The computer-implemented method of claim 23,wherein each of the one or more associations between members of the setof image annotations is determined by assigning a pair of imageannotations among the set of image annotations to a group if the pair ofimage annotations are the same.
 29. The computer-implemented method ofclaim 21, wherein each image annotation among the plurality of imageannotations comprises a box around a target feature.
 30. Thecomputer-implemented method of claim 29, further comprising applying ak-means algorithm to a centroid of the box.
 31. A system of controllingimage annotation, the system comprising: a data storage device storinginstructions for controlling image annotation; and a processorconfigured to execute the instruction to perform a method including:generating, by the processor, a plurality of image annotations ofanatomic features identified within a digital representation of imagedata; presenting, by the processor, one or more image annotations amongthe plurality of image annotations of anatomic features to a user;receiving feedback from the user regarding the one or more imageannotations; based on the received feedback, perturbing the digitalrepresentation of image data; generating, by the processor, alternativeimage annotations of anatomic features identified within the perturbeddigital representation of image data; and presenting, by the at leastone computer system, the alternative image annotations for selection orcorrection by a user.
 32. The system of claim 31, further comprising thefollowing steps prior to the step of generating a set of imageannotations of features identified within the image data: generating oneor more groups of image annotations, each group comprising one or moreimage annotations among the generated plurality of image annotations;determining a representative annotation for each group selected fromamong the image annotations of the group, the representative annotationbeing determined as an annotation among the image annotations of thegroup closest to an aggregation of the image annotations of the group;and presenting the representative annotation of each group of the one ormore groups for selection or correction by a user.
 33. The system ofclaim 31, wherein the processor is further configured to electronicallyacquire the digital representation of image data via a network.
 34. Thesystem of claim 31, wherein the processor is further configured todetermine one or more associations between members of a set of imageannotations of the plurality of image annotations, wherein each group ofimage annotations is generated based on the determined one or moreassociations.
 35. The system of claim 31, wherein the processor isfurther configured to automatically generate the set of imageannotations using multiple image analysis algorithms.
 36. The system ofclaim 34, wherein the processor in configured to determine each of theone or more associations between members of the set of image annotationsby assigning a similarity score between the members.
 37. The system ofclaim 34, wherein processor is configured to determine each of the oneor more associations between members of the set of image annotations byassigning a pair of image annotations among the set of image annotationsto a group if the pair of image annotations are the same.
 38. The systemof claim 31, wherein the processor is configured to generate a boxaround a target feature.
 39. A non-transitory computer-readable mediumfor use on a computer system containing computer-executable programminginstructions for controlling image annotation, the instructions beingexecutable by the computer system for: generating, by the at least onecomputer system, a plurality of image annotations of anatomic featuresidentified within a digital representation of image data; presenting, bythe at least one computer system, one or more image annotations amongthe plurality of image annotations of anatomic features to a user;receiving feedback from the user regarding the one or more imageannotations; based on the received feedback, perturbing the digitalrepresentation of image data; generating, by the at least one computersystem, alternative image annotations of anatomic features identifiedwithin the perturbed digital representation of image data; andpresenting, by the at least one computer system, the alternative imageannotations for selection or correction by a user. cm
 40. Thenon-transitory computer readable medium of claim 39, the instructionsbeing executable by the computer system further comprising determiningone or more associations between members of a set of image annotationsof the plurality of image annotations, wherein each of the one or moreassociations between members of the set of image annotations isdetermined by assigning a similarity score between the members.